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This checklist was prepared as an aid to evaluators, 
funding agents, clients* members of program staff about to be 
evaluated, and pertinent audiences in planning, preparing, and 
reviewing contracts for conducting evaluation studied. It contains 
neither a set of "how to" procedures for constructing contracts, nor 
a discussion of tl>e many technical, administrative, political, . legal, 
and. ethical issues concerned with evaluation contracting; it merely 
provides a list of questions and' possible options to be considered 
when dealing with evaluation contracts. Sections include: (1) . 
initiating the Contract; (2) Contract Contents; and (3) Monitoring 
and Evaluating Contract Work. In any given instance, some of the 
questions in this checklist may be either irrelevant or addressed 
through other means (e.g. standard agency procedures) and so should 
be disregarded. For example, some of the items listed in Section 2: 
Contract Contents are often included in evaluation agreements by 
reference to existing agency documents. Provided, however, is d 
comprehensive list of reminders in a format usable by both technical 
and lay groups. For those interested in more detailed discussions of 
^valuation contracting, a short reading list is 'provided at the end 
of the checklist. ( Author /GK) 
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THE CHECKLIST 

The purpose of this checklist is to aid evaluators, funding agents, 
clients, members of program staff about to be evaluated, and pertinent 
audiences in planning, preparing, and reviewing contracts for conducting 
evaluation studies. V? 

This checklist contains neither a set of "how to" procedures for 
constructing contracts, nor a discussion of the many technfcal, admini- 
strative, political, legal, and ethical issues concerned with evaluation 
contracting. This , document merely provides a list of questions and 
possible options to be considered when, dealing with evaluate contracts. 
In any given instance, some of the questions in this checklist will be 
either irrelevant or addressed through other means (e.g. standard agency 
procedures) and so should be disregarded. For example, some of the items 
listed in Section II: Contract Contents are often included in evaluation 
agreements by reference, to existing agency documents. This checklist does 
provide, however, a comprehensive "cnecklist of reminders in a format 
usable by both technical and lay groups. For those interested in more 
detailed discussions of evaluation contracting, a short wading list is 
provided at the end of the checklist. 

-XuHino ntfl, questions, or su gge sti on s f or -improvement of this ctreuldiTt 



should be directed to Nick I. Smith, Director, Research on Evaluation 
Program, Northwest Regional educational Laboratory, 300 S. W. Sixth 
Avenue, Portland, Oregon 97204. 
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I. Initiating the Contract 



NOTES 



Selection of^Evaluator 



1. 



2. 



IS AN -INTERNAL OR EXTERNAL EVALUATOR MORE 
APPROPRIATE? 

consider cost 

— credibility 
-- objectivity 

— availability of qualified personnel 



WHAT CRITERIA ARE TO BE USED IN SELECTING 
THE EVALUATOR OR EVALUATION FIRM? 

— consider formal training 
general experience 

~ specific experience with programs 
like the one to *be evaluated 

— quality and timeliness of previous 
' work 

— institutional qualifications 

— general evaluation approach or 
philosophy . * r 

— quality of submitted proposal or 
study synopses 



>HAT' PROCEDURES ARE TO BE USED TO SELECT 
JVAIUATOR OR EVALUATION FIRM? 

— consider review of persqnnel vitas 

— review of past work 

— Interviews with previous clients and 
program staff evalu ated 

— review of submitted proposal or 
study synopses (use of bl ind 
"review?) 

— use of review panel 
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Type of Contract 



I 



4. IS A CONTRACT OR FORMAL AGREEMENT REQUIRED 
OR DESIRED? 

— consider need for explicit statement 
of expectations * 
jnomtorable agreement 

— legal basis for recourse 



5. HOW WILL THE CONTRACT BE AWARDED? ^ 

Sole Source - single contractor identified 
on basis of previous knowledgu of quali- 
f i cati ons 

Synopsis Review Approach - sheltered 
compet i ti on *i n whi ch contractor i s 
selected from small group of bidders on 
basis of brief statements of proposed 
study design and bidder qualifications 

Request for Proposals (RFP) - contractor 
selected on basis of submitted proposdls 
in open competition for contract 

— consider legal constraints against 
use of sole source* or synopsis review 
procedures: 

— cost 

— known availability of qualified 
bidders, * - 

— time available to award .contract 

— possible charges of bias in selection 
process _r 
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NOTES 



HOW IS THE CONTRACT AWAROING PROCESS TO 
BE MANAGED? 

consider standard agency contracting 
pol icies^and procedures 

— appropriate institutional sign-offs 

— timing of process vis a vis tilling 
of evaluation 

« confidentiality of proposals/ 

synopses 
-- identification of bidders 

— equality of bidder communication 
(should there be a bidders confer- 
ence/) 

— use of a proposal review panel 

— consider contents of RFP or synopsis 
solicitation: 

- description of program to be 
evaluated 

- description of purpose and con- 
text of desired study 

- statement of expected evaluation 
products and available resources 

f - constraints (if any) on evalua- 
tion 

- review process and criteria for 
selecting contractor 

— consider whether the contracting 
process encourages consideration 'of 
cost-effective, high utility alter- 
natives 
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♦ Negotiation of Contract 

7. HAVE THE PRIMARY NEGOTIATORS BEEN IDENTI- 



FIED? 



— identify formal, legal : representa- 
tives of evalu&tor and funding 
agency 

— consider consultation with other 
interested and affected parties: 

- clients 

- staff to be evaluated 

- public audiences *nd citizen 
groups 



8. IS THE TIMING OF THE NEGOTIATION PROCESS 
APPROPRIATE GIVEN THE TIMING OF THE 
EVALUATION STUDY? 

— consider evaluation lead time re- 
quired for Instrument development 
sampling and treatment assignment 

— collection of background data 



9, HAVE APPROPRIALE^_PR££EDUR£S — ECU—THE — 
NEGOTIATION "PROCESS BEEN ESTABLISHED? 

— consider standard agency or evalua- 
tor procedures 

— standard legal procedures 

— confidential and non-coercive proce- 
dures 

— consider procedures for determining 
topics to be negotiated: 

- work costs 

- scope'of work 
--contract content* 

— consider mutually agreed upon time 
and resource limits for negotiation 
process 

— procedures for termination of nego- 
tiations 

— selection of alternative bidders 
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II. Contract Contents - 
Scope of Work 



1. SHOULD A STUDY PLAN OR DESIGN BE INCLUDED 
IN THE CONTRACT? 

— consider inclusion of a study plan 
which indicates the general focus, 

• activities, and products of 'tffe 
study 

consider inclusion of a detailed 
study design including* 

- purposes of study 1 

- objectives/issues/questions to 
be addressed 

- major audiences 

- intended use of results. * * 
- * - study rationale 

- assessment of program* readiness 
for evaluation , 

r * - monitoring of program implemen- 
u tat ion and impact 

- design logic and procedures 

- list of major activities 

- timeline of work 

- instrumentation to be developed 

• timing and amount of data 
collection {any site-visiting 
plans) 

- analysis and reporting 

- minimization of evaluation 
intrusiveness and reactivity 



2. SHOULD EVALUATION PERSONNEL AGREEMENTS 
BE INCLUDED IN 1HE CONTRACT? 

— consider inclusion of: names of 
actual individuals to perform the 
-evaluation study 
i - time commitments of key personnel 

- rights of personnel selection 

- rights of performance review 
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3. 



4. 



SHOULD EVALUATION PRODUCTS 8E SPECIFIED- 
IN THE CONTRACT? 

— consider inclusion of description of 
product n 

— date, location, nature of delivery 

— number of copi es 

• — k review criteria and procedures 



SHOULD THE EVALUATION REPORTING PROCE- 
DURES *BE SPECIFIED IN THE CONTRACT? 

— consider inclusion of description 
of report format and contents 

— statement of audiences . 

— anonymity of respondents 

— availability of backup documentation* 

— number of copies 

— date, location, and natjre of 
delivery • \ 

— prereview, rebuttal,' and editorial 4 
rights 

— report release rights 

— copyright and publication rights 
* — nature of reporting formats 

- memos 

- correspondence 

- technical reports 

- non-technical reports 

- news releases 

- Informal oral briefings 

- formal presentations 



act Management . 

SHOULD PROCEDURES FOR MONITORING THE 
EVALUATION WORK BE INCTOKB TN THE 
CONTRACT? 

— consider inclusion of full disclo- 
sure of evaluation procedures t 

— statement of work checkpoints and 
deadlines 

— personnel responsible for reviews 

— review criteria 

~ timeliness 

- degree of completion 

- technical quality 

- compliance with professional 
standards 



* o 

ERIC 




NOTES 



SHOULD PROCEDURES 8E SPECIFIED IN THE ' 
CONTRACT TO INSURE ALL NECESSARY PRQJEC - 
TlON§>, * } ( ; t? 

— consider inclusion of provisions 
! * covering conflicts of interest 

non-exploitation of parties 
— protection of human subjects 

— forms clearance 

*•— informed consent and rignt of 
respondent withdrawal without 
I penalty * 
— ' nondiscrimination fl , 

— compliance with pertinent laws 

— confidentiality and protection of 
privacy _ f 

--'^ata security ^ /f^ 
insurance/bonding of contractor 

SHOULD, DETAILS CONCERNING FINANCIAL PAY- 
MENT BPIN^LUDED IN THE fONTRACT? f 0 

— consider inclusion of. itemized 
project budget,^ { 

— "statement • pf ^bunt, timing, L and f 
' nature of payments (fixed price or 

. cost reimbursement) 
conditions of payments v 
- contributed resources (computer 
time, secretarial support) 

— hidden or unexpected costs (dupli- } 
' cation of final report) 

— restrictions on chargeable items L 

— overhead rates • 
default penalties 



SHOULD PROVISIONS 8E INCLUDED IN THE 
CONTRACT FOR AMENDING OR ADJUDICATING 
GRIEVANCES UNDER THE AGREEMENT? 

— consider inclusion of statement of 
conditions and procedures foramend- 
ing contract 

— criteria and procedures for deter- 
mining contract compliance 

-- procedures for conflict resolution 

— forma* grievance adjudication proce- 
dures *~ » 

— rights of appeal- and redress 
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NOTES 

9. SHOULD OTHER MISCELLANEOUS AGREEMENTS -BE 
INCLUDED IN THE CONTRACT? 

— consider inclusion of agreements on 17 
site access and data access 

— additional restrictions, if any, on 
evaluation staff or other parties 
to the agreement 

— agreements concerning use of results 

— who owns the* data, how .long it 
should be retained and in what form 

— " use of subcontractors 

v agreements on contributed efforts or 
actions of nonevaluation parties to 
agreement 

— agreements concerning use of study 
for" ancillary research or policy 
work 

— other evaluator responsibilities 

— other client or funding agent 
responsibilities 
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III. Monitoring and Evaluating Contract Work 
Monitoring Work 



1. HAVE PROCEDURES FOR PROPER MONITORING OF 
THE CONTRACT WORK BEEN SPECIFIED AND 
IMPLEMENTED? 

— see Section 1 1.5 

consider whether periodic management 
reports are required 

- procedures for charting work 
progress 

- problem notification procedures 

- mechanises for preservation of 
client control 



2. HAVE PROCEDURES BEEN DEVELOPED FOR MODI- 
FYING WORK OR CONTRACT BASED ON DEVIATION 
FROM PLAN? 

. — see Sections II. 2, II. 7, II. 8 



Evaluating work 



3. HAVE FINAL REVIEW PROCEDURES BEEN- ESTAB- 
LISHED TO EVALUATE THE QUALITY OF THE 
COMPLETED. WORK? 

— see Sections II. 7, II. 8 

-- consider use of independent review 
panels 

— application of professional stan- 
dards 

formal critiques of the evaluation 
study (meta-evaluation) 
-- re-analysis of evaluation data 
(secondary iJata analysis) 
public review of procedures and 
findings (adversarial or community 
hearings) 



- consider procedures for nonpenalized 
admission of error 

- self-criticism uy contract parties 



HAVE OPTIONS FOR RECOURSE BEEN EXPLORED 
SHOULD WORK NOT COMPLY WITH CONTRACTUAL 
AGREEMENTS? 

— consider informal letters, of inquiry 
--TormaT letters of complaint 

negotiated settlements 

— nonpayment of funds 

— refusal to reemploy 

— legal suit 

~ public exposure 

consider type of recourse' appropri- 
ate for each level of deviation frofl{ 
agreement 
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